Robot Guidance by Human Pointing Gestures

In the search for intuitive man-machine interfaces the visual recognition of human gestures would provide a powerful means to guide robot movements. In our paper we report on the development of the modular neural system \See-Eagle" for the visual guidance of robot pick-and-place actions. Several neural networks are integrated to a single system that visually recognizes human hand pointing gestures from stereo pairs of color video images. The accuracy of the pointing recognition system alone allows to estimate the pointing target to an accuracy of 2 cm in a workspace area of 50 50 cm. The output of the hand recognition stage is further processed by a set of color-sensitive neural networks to determine the cartesian location of the target object that is referenced by the pointing gesture. Finally, this information is used to guide a robot to pick the target object and place it at another location that can be speciied by a second pointing gesture. The accuracy of the combined system allows to identify the location of the referenced target object to an accuracy of 1 cm in a workspace area of 50 50 cm. In our current environment, this is suucient to pick and place arbitrarily positioned target objects within the workspace. The rapidly developing technology in the elds of robotics and virtual reality requires the development of new and more powerful interfaces for connguration and control of such devices. These interfaces should be intuitive for the human operator and comfortable to use. Practical solutions so far require the operator to wear a device that can transfer the necessary information. One typical example is the data glove 21, 19]. Clearly, in the long run solutions that are contactless will be much more desirable, and vision is one of the major modalities that appears especially suited for the realization of such solutions. In the present paper, we focus on a still restricted but very important task in robot control, the guidance of robot pick-and-place actions by unconstrained human pointing gestures in a realistic laboratory environment. The input of target locations by pointing gestures provides a powerful, very intuitive and comfortable functionality for a vision-based man-machine interface for guiding robots and extends previous work that focused on the detection of hand location or the discrimination of a small, discrete number of hand gestures only 14, 1, 2, 11]. Besides two color cameras, no special device is necessary to evaluate …